(1) Field of the Invention
The present invention is directed to the technical problem of synchronizing the processing and distribution of audio-visual assets in a distributed network of asset providers and processors, and more particularly to the technical problem of the synchronization and processing of live broadcast television, including real-time implementation of match moving computer graphics technology, in a distributed, or cloud based, production environment.
(2) Description of Related Art
Television production is the technical art of transforming an event into video images, preferably with an audio accompaniment, for display to an end viewer. Inherent in this technical art is the problem of time synchronization of audio-visual assets originating from different sources.
A traditional solution to this problem is to “genlock” all the television production devices, i.e., arrange for all the equipment being used for the production to function off, or be locked to, a single signal generator. This master signal generator then acts as a common clock so that all the signals originating from the genlocked devices have a common start time and the same phase and may, therefore, be easily combined together.
This solution, however, has its limitations. In particular, it may become cumbersome and inefficient when the production is not performed in a single studio, but, as is increasingly common, is undertaken using a distributed, or cloud based, environment in which the video of the event and the graphics being incorporated into them may be provided from remotely located sources and nodes. A reliable, broadcast quality output may be difficult to achieve because of, for instance, unpredictable variations in transmission times between the various nodes and sources in the system.
In the field of live sports broadcasting it has become common practice to use match moving technology to superimpose graphics on live events so that they look as they are a part of the original scene being broadcast. Using this technology exacerbates the problem of synchronization, especially when efficiencies could be obtained by splitting up the match moving tasks and assigning each of them to a different one of the distributed nodes. The additional problems introduced by this approach include variations in processing times as well as the variations in node to node transmission.
The system and method of the present invention provide a robust, workable solution to such problems as is described in detail below.
The relevant prior art addressing the technical problem articulated above includes:
U.S. Pat. No. 9,116,220 issued to Liu et al. on Aug. 25, 2015 entitled “Time synchronizing sensor continuous and state data signals between nodes across a network” that describes techniques for synchronization of sensor signals between devices. One or more of the devices may collect sensor data. The device may create a sensor signal from the sensor data, which it may make available to other devices upon a publisher/subscriber model. The other devices may subscribe to sensor signals they choose. A device could be a provider or a consumer of the sensor signals. A device may have a layer of code between an operating system and software applications that processes the data for the applications. The processing may include such actions as synchronizing the data in a sensor signal to a local time clock, predicting future values for data in a sensor signal, and providing data samples for a sensor signal at a frequency that an application requests, among other actions.
US Patent Application no. 20130067116 published by Brian H. Ostergren on Mar. 14, 2013 entitled “Synchronizing Streaming Video between Viewers over a Network” that describes a technology by which machines such as gaming (including media) systems are configured to synchronize streaming video between peers over a network, including participants communicating via voice or video chat. The video synchronization may include “full” synchronization that operates to synchronize streaming video that is currently unsynchronized video, by causing participants to stop and buffer, and resume playing from a specified position in the stream. Video synchronization also may include “fast” synchronization that operates to re-synchronize video that has begun to drift out of synchronization, such as by briefly pausing any stream that has gotten ahead of others. Also described is replicating the activation of a video transport control (e.g., pause, fast forward, rewind, skip, and the like) that occurs on one machine to other machines, such that video players behave as if there is a common remote control among the peer systems.
US Patent Application no. 20060149850 published by Timothy D. Bowman on Jul. 6, 2006 entitled “Method and apparatus for synchronizing playback of streaming media in multiple output devices” that describes a method and apparatus for synchronizing streaming media with multiple output devices. One or more media servers serve media streams to one or more output devices (i.e., players). For playback synchronization, one output device is the “master”, whereas the remaining output devices are “slaves”. More data is requested from the media server by the “master” device to maintain a nominal buffer fill level over time. The “slave” devices receive streamed data from the media server at the rate determined by the master device's data requests, and the average rate of data flow over the streaming network is thus controlled by the frequency of the single “master” device's crystal. “Slave” devices make playback rate corrections to maintain respective buffer fill levels within upper and lower threshold levels. For slow networks, each media data packet timestamp is calculated from the time the master's buffer reaches nominal level.
U.S. Pat. No. 5,264,933 issued to Rosser et al. on Nov. 23, 1993 entitled “Television displays having selected inserted indicia”. The patent description includes basic concepts of match move video insertion technology and is hereby incorporated by reference in its entirety.
The application describes an apparatus and method of altering video images to enable the addition of images, message, slogans or indicia in such a manner that they appear to be part of the original image as displayed. The application of the apparatus and method is particularly adapted to be used as advertising during sporting events or other events whereby an operator selects a portion of the video image display such as a portion of a tennis court, recognizes the selected portion and inserts a logo or advertising message into that selected portion. The message is inserted into the selected portion of the court independent of how the scene is being panned or viewed by the television camera. The apparatus employs a pattern recognition algorithm such as the Burt Pyramid Algorithm to recognize the specific selected image portion and by means of the image processor substitute the logo into the first image to provide a display which appears as if the inserted image was actually painted on or otherwise permanently positioned on the court. Thus the inserted images are seamlessly and realistically incorporated into the original even as the original scene is moved, panned, magnified, zoomed or otherwise altered in size or perspective. The resulting composite image appears to the television viewer to be absolutely real and present at the scene.
Various other implementations are known in the art, but, as far as can be reasonably ascertained from currently and readily available documentation, fail to adequately address all of the problems solved by the invention described herein.